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Abstract 

Background: Patient decision aids (PtDA) are developed to facilitate informed, value-based decisions about health. 
Research suggests that even when informed with necessary evidence and information, cognitive errors can prevent 
patients from choosing the option that is most congruent with their own values. We sought to utilize principles of 
behavioural economics to develop a computer application that presents information from conventional decision 
aids in a way that reduces these errors, subsequently promoting higher quality decisions. 

Method: The Dynamic Computer Interactive Decision Application (DCIDA) was developed to target four common 
errors that can impede quality decision making with PtDAs: unstable values, order effects, overweighting of rare 
events, and information overload. Healthy volunteers were recruited to an interview to use three PtDAs converted 
to the DCIDA on a computer equipped with an eye tracker. Participants were first used a conventional PtDA, and 
then subsequently used the DCIDA version. User testing was assessed based on whether respondents found the 
software both usable: evaluated using a) eye-tracking, b) the system usability scale, and c) user verbal responses 
from a 'think aloud' protocol; and useful: evaluated using a) eye-tracking, b) whether preferences for options were 
changed, and c) and the decisional conflict scale. 

Results: Of the 20 participants recruited to the study, 1 1 were male (55%), the mean age was 35, 18 had at least a 
high school education (90%), and 8 (40%) had a college or university degree. Eye-tracking results, alongside a mean 
system usability scale score of 73 (range 68-85), indicated a reasonable degree of usability for the DCIDA. The think 
aloud study suggested areas for further improvement. The DCIDA also appeared to be useful to participants 
wherein subjects focused more on the features of the decision that were most important to them (21% increase in 
time spent focusing on the most important feature). Seven subjects (25%) changed their preferred option when 
using DCIDA. 

Conclusion: Preliminary results suggest that DCIDA has potential to improve the quality of patient decision-making. 
Next steps include larger studies to test individual components of DCIDA and feasibility testing with patients 
making real decisions. 
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Background 

In recent years, numerous patient decision aids (PtDA) 
have been developed to facilitate informed, value based 
decisions about treatment options [1]. They have been de- 
veloped in response to many beneficial treatments or 
screening strategies which also have negative aspects such 
as side-effects or high costs. What is best for one patient 
may be different to another depending on how each values 
the attributes of each option [2]. Health professionals are 
often poor proxies of patients' values [3,4], and often fail 
appropriately 'diagnose' patient preferences [5]. Patients 
can also have unrealistic expectations of treatment bene- 
fits and harms [6] . 

PtDAs provide facts about the condition, options, and 
attributes such as outcomes and probabilities for each op- 
tion; a value clarification task that helps patients evaluate 
which attributes matter most to them; and a guide in the 
steps of deliberation and communication required for the 
informed patient to reach their goal - concordance be- 
tween what matters most to them and their chosen option 
[7]. An updated Cochrane systematic review found that, 
among 115 studies involving 34,444 participants, PtDA in- 
crease patients' knowledge about treatment options, and 
reduce their decisional conflict related to feeling unin- 
formed and unclear about their personal values [1]. 

There have been tremendous advances in the way 
PtDAs are developed, from the way risks are presented 
[8], to the use of animated stories to better communicate 
information [9]. However, there has been comparatively 
little research on reducing decision errors in people 
using PtDAs [10]. While in theory, PtDAs should help 
patients identify the best treatment option for them, re- 
search shows that various cognitive biases may result in 
errors that prevent this in some situations [11-14]. For 
example, individuals are known to make different choices 
when their options are framed as gains or losses, prefer- 
ring a surgical procedure with a 90% survival rate to one 
with a 10% mortality rate [15]. Studies have shown that in- 
dividual treatment choices are unduly influenced by 
whether individuals learn first about potential harms or 
potential benefits [10], and individuals are intimidated and 
overwhelmed by options that include numerous rare side- 
effects leading to irrational decisions [16]. 

The objective of this study was to develop and user test 
a computer application that enhances conventional PtDAs 
to improve the quality of decisions by helping patients 
overcome common decision errors. 

Development of DCIDA 
Theoretical motivation 

Normative decision theory suggests that for patients to 
approach treatment or screening decisions rationally 
they need to weigh benefits and harms using deliberative 
"compensatory strategies" to make trade-offs [17]. That 



is, a patient can "compensate" for the negative feature of 
one option by considering a positive feature. A patient 
looking at cancer screening options may not want to 
have annual testing, but would not mind if such frequent 
testing were non-invasive. While this approach helps 
people identify the treatment option that matches their 
informed values, descriptive decision theory has identi- 
fied numerous errors in peoples' decisions caused by 
cognitive biases and simplifying heuristics [12-15]. An 
understanding that people have two systems for cogni- 
tive functioning has provided a framework for under- 
standing these errors and providing effective strategies 
for improving decision making [18]. System 1 refers to 
people's intuitive system, which is typically fast, auto- 
matic, effortless, implicit, and emotional. While often 
useful for simple decisions, they can lead to decision er- 
rors for more difficult decisions, such as ones requiring 
compensatory strategies. System 2 refers to reasoning 
that is slower, conscious, effortful, explicit, and logical. 
Recent research suggests that when faced with decisions 
and information that is unfamiliar, complex, or over- 
whelming - all common traits targeted by most PtDAs - 
people can switch to use System 1 functioning, which can 
lead to decision errors [12,13]. 

By reviewing PtDAs contained in the Ottawa repository 
[19], the project team identified few examples where PtDAs 
helped patients making trade-ofis, and beyond simple value 
clarification exercises gave patients little help in choosing 
what was best for them. The team identified four issues 
common to nearly all PtDAs that were believed could im- 
pede the quality decision-making, and for which interven- 
tions were feasible: unstable preferences, order effects, 
overweighting of rare events, and information overload. 

i) Unstable values: Unlike most goods and services, 
where markets help form stable values and 
consumers understand the relative value of 
attributes through a process of trial and error and 
notion of sacrifice, evidence indicates that patients 
often have unstable values when it comes to health 
care [20,21]. This follows from the fact that the 
potential benefits and harms of treatment may be 
unfamiliar to patients and impossible to evaluate 
without a great deal of information and reflection. 
The consequence is that people's decisions may be 
inconsistent with their true underlying values. 

ii) Order effects: For a rational decision maker, the way 
information is presented should not have an 
influence on the choice that is made. However, there 
is evidence that people tend to, for example, 
remember information presented to them first 
(primacy effect) or last (recency effect) [22-24]. The 
decision error in PtDAs caused by this heuristic is 
demonstrated in a study of women at high risk of 
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breast cancer who were considering tamoxifen, which 
found that patients who learned first about the risks of 
the drug thought more favourably of tamoxifen than 
patients who learned first about the benefits [10]. 
Nevertheless, harms and benefits must be presented 
in some order in a PtDA, and therefore, the designer 
of the PtDA may inadvertently influence the patient to 
choose a given option. 

m) Overweighting of rare events: Most treatments have 
multiple complications. PtDAs are expected to 
inform patients about any treatment complication 
that is reasonably likely to occur. Although there are 
no absolute criteria regarding which complications 
must be included for meeting informed consent, 
most PtDAs show information on any moderately 
severe complication that occurs at least 1% of the 
time, and serious complications that occur even less 
often than that. Prospect theory describes how people 
systematically overweigh small probabilities in terms 
of their impact on decisions. The consequence of this 
phenomenon is that patients using PtDAs can be 
scared away from treatments with multiple rare 
side-effects that rationally would have appeared the 
best treatment option for them [14,16]. 

iv) Information overload: When individuals receive 
conflicting, incomplete, uncertain, or excessive 
information, they experience ambiguity and can 
make contradictory decisions [25]. The role of 
information overload causing ambiguity in investment 
decision-making has been well documented. When 
the complexity of decision-making increases, people 
tend to expend less effort to actually make their 
decision, seek others to make decisions for them, 
or select default options if available [26] . 

A number of promising strategies have been uncov- 
ered for overcoming specific decision errors. One ap- 
proach is to encourage people to use System 2 thinking 
instead of System 1 by making the information and de- 
cision less overwhelming. This can be achieved by fo- 
cussing attention on the most pertinent information, 
and by using analytic processes which reduce the num- 
ber of 'internal calculations' which require cognitive ef- 
fort [27-30]. In the area of PtDAs, the predominant 
approach has been to employ formal decision analysis 
techniques which quantifies patients' values and inte- 
grates them with probabilistic information [31-33]. 
While there are many perceived advantages to this pre- 
scriptive approach, there are also criticisms. First, 'opti- 
mal' options derived from decision analysis are reliant 
on assumptions, theories and inaccuracies in inputs 
which mean they may not actually prescribe the best 
course of action for each patient [34]. Second, the 
current approaches to decision analysis are typically 



'overt' to be best course of action, and consequently have 
been argued to be an extension of paternalism, comprom- 
ising patient autonomy [35]. 

An alternative strategy is to leverage System 1 thinking 
by changing the decision environment to maximize the 
odds that people will make high quality decisions given 
known biases [26]. For example, it is known that most 
people have a bias towards inaction, in which providing a 
default option has been found to be a powerful decision 
enhancer [36]. These strategies have increasingly been re- 
ferred to as 'nudgesi reflecting the unavoidable paternalis- 
tic role of the designer of the tool (in this case, PtDAs) in 
influencing users' choices [37]. Nudges have been defined 
as "...any aspect of the choice architecture that alters peo- 
ple's behavior in a predictable way without forbidding any 
options or significantly changing their economic incen- 
tives..." [37], and so seek to preserve patient autonomy 
and freedom of choice. 

We sought to employ decision analysis more covertly 
[38] to improve decision-making by testing various 'nudges' 
which help people focus on the information and options 
that reflect their values, and simplify their trade-offs. 

Features 

We developed a dynamic computer interactive decision 
application (DCIDA - pronounced 'decider') to employ 
some of the strategies described above. The overarching 
aim of the DCIDA is to present information and the de- 
cision to each person in an individualized way in order 
to maximize their ability to make choices that reflect 
their own informed, stable values. Acknowledging that it 
is rare that any decision support system will induce opti- 
mal decisions, the goal is to improve the quality of deci- 
sions, from what would be made with conventional 
PtDAs. Multi-criteria decision analysis (MCDA) using a 
weighted additive model motivates the application. For 
treatment decisions, this assumes the preferred option is 
based on the sum of the importance or weight (on a 0- 
100 scale) of each attribute, say benefit or harm, multi- 
plied by each option's score (on a 0-100 scale) for that 
attribute. The treatment with the highest weighted score 
or expected value indicates the patient's optimal option. 

The application contains the same content and informa- 
tion as a PtDA, explaining the condition, providing infor- 
mation about options and their characteristics (benefits, 
side-effects, costs etc.) using probabilities and pictographs 
to describe baseline and incremental absolute risks where 
appropriate, a value clarification exercise, and a summary 
of information to help the patient deliberate on the deci- 
sion along with an opportunity to select the preferred op- 
tion. However, the way this content and information is 
structured and organized differs, and where possible indi- 
vidualized to make it simpler for each person to choose 
what is best for them. Figure 1 compares the pathway a 
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The summary shows 
criteria ordered by 
importance. The size of 
row is in accordance to 
the weight (e.g. side- 
effect A represents 40% 
of the total height). The 
colours represent the 
scores for each criteria, 
where white is 
preferred over dark 
grey. The default is pre- 
selected by calculated 
expected value 



There is no 
default option 



Conventional PtDA 



DCIDA version of the PtDA 



Introduction: Information about the 
decision aid, the condition, and the 
decision being considered 




Information about the options: benefits and harms 

including pictographs for risk information 
Benefit 

When using option 1, your pain will be relieved 
immediately 

Side-effect A 




Summary information: 



Criteria 


Opt/on 




2 


3 


Benefit 


Smali 
improvement 


Medium 
improvement 


Large 
improvement 


Side-effect A 


Low risk 


Medium risk 


High mk 


Side-effect B 


Low risk 


Low risl< 


Medium risk 


Side-effect C 


Low risk 


Medium risk 


Medium risl< 


Mode of 
administration 




Pili 


injection 




Value clarification exercise: 
Matters 

Not 



Benefit 
Side-effect A 
Side-effect B 
Side-effect C 
Mode of admin 



Preferred option: 





Option 


1 1 2 1 3 


Preferred option? 


o 1 o 1 O 



Value clarification exercise: 



Matters 
Not A lot 



Benefit 
Side-effect A 
Side-effect B 
Side-effect C 
Mode of admin 



Information about the options and scoring: benefits 
and harms including pictographs for risk information 

Side-effect A 

When using option 1, about 2 in 100 (4%) fewer people 
like you (3 in 100 total) will experience side-effect A in 
the next 5 years compared to no treatment 



How important is this to you? 



~l l Somewhatll ~1( Very | 



Summary information and preferred option: 



Criteria 


DpWon 


1 


2 


3 


Side-effect A 




Medium risk 


High risk 


Side-effect B 


Low risk 


Low risk 


Medium risk 


Benefit 


Smaii 
improvement 


Medium 
improvement 


Large 
improvement 


IMode of administration 




Piil 


Injection 


Side-effect C 


Low risk 


Medium risk 


Medium risk 


Preferred option? 




O 


O 



Preparation for decision mai<ing 
and next steps: 



Figure 1 Different pathways for obtaining information and indicating preferences for DCIDA vs conventional PtDA. 



patient would take between a conventional and DCIDA 
version of a PtDA. For example, the first unique feature of 
the DCIDA is that in step 1 the value clarification task, 
which is usually near the end of a conventional PtDA, is 
moved to the beginning. The objective of the task was to 
a) provide an opportunity for individuals to reflect on the 
relative value of attributes and subsequently derive more 
stable values (see "i) Unstable values" above) and b) to 
generate the weights for each attribute for use later in the 
tool. After preliminary testing, we decided to use an 



interactive form of constant sum exercise (also known as a 
"budget pie") which requires users to allocate a certain 
number of points (often 100) to each attribute in accord- 
ance with the relative importance of each. Constant sum 
exercises have a long history of use and incorporate a 
number of properties desirable for encouraging compen- 
satory decisions [39], but have been criticized for requiring 
a higher levels of numeracy [40]. We developed a simpler 
version that requires users to move multiple sliders, all 
linked to an interactive pie chart (Figure 2). For individuals 
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Matters 



Not 



A lot 



Decreased risk of death due to lung cancer 

Lowering of white blood count 

Fatigue which limits activity 

Nausea with or without vomiting 

Risk of vein irritation 

Appetite loss/Stopping eating 

Lowering of the red blood count 

Fever and low white blood count 

Complete hair loss 

Constipation/Bowel Movement <3 times/wk 
Numbness in hands or feet Hearing loss 



Coosiip«i<xi/Bo 
w*l Movtm«nt 




whiit blood 

count 



RiskoOo^n Nause.w.ftor 
3K 



In this task, information on the range of outcomes for each attribute is first described. The sliders for all attributes begin in the middle. Users can move each slider to indicate its 
importance. The other sliders do not move, but the area in the pie chart indicates its relative importance. The key feature is that it helps people understand that indicating all 
attributes 'matter a lot' infers they have equal relative value. 

Figure 2 Example of constrained interactive pie chart. 



with low numeracy, pie charts can be an effective format 
for communicating the "gist" of health knowledge and 
treatment choices [41]. As participants navigated through 
the following steps of the DCIDA version, they could re- 
turn at any time to the budget pie exercise to change the 
weights allocated to different attributes of the decision. 

After the values clarification task, Step 2 presented par- 
ticipants with detailed information (e.g. the actual risk) for 
each attribute. Where possible, the information was ex- 
plained in a simple format, including pictographs where 
necessary to communicate risks. Importantly, this infor- 
mation was ordered in accordance with the rank of the in- 
dividual's weights - obtained from the value clarification 
exercise. For each attribute, the patient was asked to rate 
how important the attribute (outcome such as benefit or 
harm) for each option. This was used to create the score 
for each attribute. In our preliminary testing we used a 
simple 5-point scale to derive numerical scores. 

In step 3, the summary information for all consequences 
was displayed. In contrast to a conventional PtDA: 1) the 
consequences were ordered in accordance with the rank 
of the individual's weights - obtained from the value clari- 
fication exercise. This aimed to exploit order effects by 
nudging individuals to focus on the information that 
would most significantly influence their decision (see "ii) 
Order effects" above); 2) rows were further sized in propor- 
tion to the weights of each consequence, with the most 
important consequences being presented in wide rows 
and less important consequences in narrow rows. This 
served to take attention away from rare events for the ma- 
jority of people who rated these attributes to have low 



importance in the value clarification exercise (see "iii) 
Overweighting of rare events" above); 3) for each conse- 
quence, the colour for each option was based on the score, 
with a lighter shade of grey indicating a more preferred 
option. Colouring aimed to simplify the information pre- 
sented (akin to traffic light labelling for the nutritional of 
food [42]), enabling individuals to process multiple pieces 
of information and distinguish between harms and bene- 
fits (see "iv) Information overload" above); 4) the sum of 
the weights and scores were used to determine which op- 
tion would be preferred using MCDA (Figure 3). This in- 
dicated the 'optimal' choice for a given individual and 
became the default option for the participant, helping 
overcome information overload. On the summary page, 
users were able to select an option other than the default 
optimal choice; however the presence of a selected default 
option has proven to help overcome ambiguity [36] . 

Development 

Three PtDAs available to the developers were transformed 
into DCIDA versions. The goal was to ensure the DCIDA 
was usable for a broad range of PtDAs, each of which has a 
different set of individual characteristics. The first PtDA 
used information on newly developed over the counter 
medication choices for the treatment of knee osteoarthritis, 
where it is suspected patients underestimate the risks asso- 
ciated with high doses. The options were no treatment, 
acetaminophen (lower benefit, but fewer side-effects), and 
non-steroidal anti-inflammatory drugs (NSAIDs) (greater 
benefit, but greater side-effects). The second PtDA focussed 
on treatment options for patients with Obstructive Sleep 
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Options 



Benefit (0.2) 




Mode of administration (0.1) 


Attributes (weights) 


Sidc-cffcct A (0.4) 




Side-effect B (0.2) 




Side-effect C (0.1) 


Total Score (1.0) 



12 3 


1 




1 


Small improvement 
(0.1) 




Medium improvement 
(0.3) 




Large improvement 
(0.6) 


1 


1 




None 
(0.7) 




Pill 
(0.2) 




Injection 
(0.1) 


1 




1 


I,o\v risk 
(0.5) 




Medium risk 
(0.4) 




1 ligh risk 
(0.1) 


1 


1 


1 


Low risk 
(0.4) 




Low risk 
(0.4) 




Medium risk 
(0.2) 


1 


1 


1 


Low risk 
(0.6) 




Medium risk 
(0.2) 




Medium risk 
(0.2) 


1 1 1 
0.43 0.34 0.23 



AiCDA: Multi-crileria decision analysis 

DCIDA: Dynamic Computer Interactive Decision Application 

This individual rates side-effect A to be the most important (40%) folhwed by the benefit (20%), side-effect B (20%), and other attributes (where the mights add up to 100%). 
They score the large improvement in benefit achieved by option i highly (0.6) but this only contributes 0. 12 (0.6x0.2) to the overall score. 'The total score is simply the sum of the 
weighted scores and suggests that option 1 is preferred (43%), followed by option 2 (34%), and finally option 3 (23%). 

Figure 3 Example of conventional MCDA weights and scores, and total scores and DCIDA. 



Apnea (OS A). The primary treatment option is continuous 
positive airway pressure (CPAP), which is effective but in- 
convenient to use. Less invasive alternatives such as oral ap- 
pliances or no treatment are also options. The third PtDA 
considers chemotherapy options for patients with late stage 
non-small cell lung cancer (NSCLC). Treatments with 
higher efficacy are also associated with more firequent and 
severe side-effects. All three PtDAs were conceived using 
the Ottawa Framework [43] and were reviewed to Inter- 
national Patient Decision Aid Standards (IPDAS) guidelines 
[7]. DCIDA versions of each PtDA were made using exactiy 
the same information. An example of the DCIDA version 
of the PtDAs is available at: http://dcida.cheos.ubc.ca/osal. 

Methods 

Overview 

We focussed our user testing on both usability (whether 
the user can do what they want to do to without hin- 
drance, hesitation, or questions) and usefulness (does it 
help the user mal<e a better decision) [44]. Two common 
approaches to user testing were used - eye-tracking and a 
thinl< aloud protocol - and these were supplemented with 
various validated questionnaires. Ethics was granted from 
University of British Columbia Ethics Board. 

Participants and procedures 

A sample of healthy, English-speaking volunteers was 
recruited through online advertisements and posters. 



Participants were seated at a computer equipped with 
an eye-tracker and the interviewer explained the pur- 
pose of the study. After gaining consent, the participant 
went through the calibration procedure to initialize the 
eye-tracking system. The participant was then asked to 
choose which clinical scenario they wanted to imagine 
they were facing: Knee Osteoarthritis, OSA, or NSCLC, 
and proceed to complete an online version of a conven- 
tional PtDA followed by the DCIDA version created for 
their chosen scenario. 

Figure 1 shows the flow for the conventional PtDA. After 
selecting their preferred option, they were asked to indicate 
their uncertainty in their decision by completing the Deci- 
sional Conflict Scale (DCS). They then went through the 
DCIDA version (right side of Figure 1) and were again 
asked to choose their preferred option, and their uncer- 
tainty as measured by the DCS. Finally, they were asked to 
complete the System Usability Scale (SUS) focussing on the 
DCIDA. After completing the tool, an interview was used 
to discover users' impressions about the tool and their ex- 
periences. Upon completion of the tasks, the user was com- 
pensated with a $25 dollar gift certificate. 

Eye-tracking 

Eye tracking is a promising method for usability testing 
since it can evaluate individuals' information processing 
while they deliberate on decisions [45] . It makes it possible 
to determine what type of information individuals look at 
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and to what extent information is processed. The eye- 
tracker method is widely used in marketing research as 
well as research on cognitive processing and decision- 
making processes [46] . 

When individuals look at information while reading or 
searching, they continually make rapid eye movements 
called saccades. Between saccades, the eyes remain rela- 
tively still for approximately 200-500 ms [47]. Individuals 
do not obtain new information during a saccade because 
the eyes are moving too quickly. Rather, higher levels of 
information processing require deeper cognitive processes 
that can only be processed while the eyes are fixated. Re- 
search on information processing is therefore mainly con- 
cerned with fixation durations [47] . Fixation durations are 
assessed by an infirared camera system built into the eye 
tracker. This camera measures the light reflex on the cor- 
nea of individuals sitting in front of the computer screen. 

In this study, the eye-tracking data was used for two pur- 
poses. First, to assess user experience, we analyzed heatmaps 
of each page of the tool to ensure respondents were consist- 
entiy looking and reading the important aspects of design 
such as instructions. Heatmaps visually display the areas in 
which fixations on each page occur. Second, to assess use- 
fulness we compared fixations between the conventional 
display versus the DCIDA display. To analyze the eye- 
tracker data, we subdivided areas of the summary screen 
into areas of interest based on each attribute. The time indi- 
viduals spend looking at relevant information in relation to 
the total time needed to look at the whole summary infor- 
mation was used to indicate attributes participants were 
spending time deliberating on [47]. It is expressed by the 
relative fixation duration, that is, the percentage of the time 
spent fixating on each attribute relative to the time spent 
looking at the whole summary information [47]. A Tobii 
T120 eye tracker embedded in a 17" display was used. 

Decisional conflict scale (DCS) 

After stating a preference for one treatment, participants 
were asked to evaluate their uncertainty in their decision 
based on a subscale of the DCS [48]. The DCS is a vali- 
dated scale that assesses patients' conflict and uncertainty 
in their decision. While the full scale comprises 16 items di- 
vided into 5 subscales - uncertainty, inadequate knowledge, 
values clarity, lack of support, and ineffective choice - we 
focussed simply on the uncertainty subscale, the compo- 
nent the DCIDA attempts to increase confidence in the 
decision. This subscale includes three items: how clear the 
patient is about the best choice, how sure they feel about 
that choice, and how easy the choice was to mal<e. All 
items are reported on a 5 point Likert scale from strongly 
agree to strongly disagree. Lower scores are desirable as 
they indicated less conflict. An effect size of 0.06 to 0.3 
has been reported to discriminate between decision sup- 
porting interventions [49] . 



System Usability Scale (SUS) 

After completing the tool, participants were asked to an- 
swer an adapted version of the SUS. This validated scale 
asks 10 questions about aspects of user friendliness, con- 
tent integration, and support needed to answer the tool 
providing a score between 0 and 100 [50]. The SUS is a 
commonly used quantitative assessment of usability. It is 
useful for rough comparison purposes, including assessing 
the effects of changes from one prototype iteration to the 
next, and for drawing preliminary conclusions about over- 
all usability of a system. To our knowledge there is no 
established SUS threshold for usability, however previous 
studies have shown that a SUS score above 68 is above 
average for all studies that have used the scale, while a 
score above 74 would place the system in the top 30% 
[51]. We chose to adopt these thresholds for our study. 

Think aloud study 

We used a verbal protocol analysis, a form of 'think aloud' 
technique, to further investigate respondents' choices [52] . 
Think aloud data can be obtained in two ways: concurrent, 
where respondents are asked to verbalize their thoughts as 
they complete a task, and retrospective, where respondents 
are asked to describe what they were thinking after the 
task was completed. Following experience from previous 
studies, we used a hybrid approach whereby respondents 
were asked to think aloud as they completed the tool, 
however if they did not think aloud for a period of 10 sec- 
onds, the interviewer would ask them to reflect back on 
their choices [53]. This approach interferes less with re- 
spondents' thought processes while still allowing an ex- 
ploration of how respondents were making choices. 
Respondents were asked not to explain or plan what they 
were saying, but to act as if they were speaking to them- 
selves. Following the survey, the interviewer asked respon- 
dents debriefing questions. In general, respondents were 
asked how they found the information and choices they 
were presented with and how they would improve the 
tool. The interviews were tape-recorded and later tran- 
scribed. Responses were coded by each step in the tool 
and whether they were related to user experience or us- 
ability. Two independent reviewers then coded the valence 
of each comment (e.g. positive, negative or neutral) and 
differences resolved by discussion. 

Results 

Participants 

In total, 20 participants were recruited via posters and 
completed the study. Eleven participants were male (55%), 
a 15 participants were white, and the mean age was 35 
(range 19-59) (Table 1). All but two participants had at 
least a high school education and 8 (40%) had a college or 
university degree. None of the participants were suffering 
from a serious illness and only three participants were 
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Table 1 Participant characteristics 



Value 



Age, mean (range) 
Sex, % male 
Race, n% 
White 
Asian 

Education, n % 

At least higii sciiool 

At least college or university graduate 



35.2 (19-59) 
55% 

1 5 (75%) 
5 (25%) 

1 8 (90%) 
8 (40%) 



currently taking prescription medications. Twelve respon- 
dents chose to complete the OSA version and four each 
chose the NSCLC and Osteoarthritis versions. 



User experience 
Eye-tracking 

In general, heat maps suggested participants were reading 
all the relevant information on each page. For the first 4 
participants, it was noted that there were few fixations on 
the titles of the scales of the value clarification task 
(whether each attribute was more or less important). The 
titles were increased in size and bolded, and this led to in- 
creased fixations in subsequent participants. 



sus 

The mean SUS score for DCIDA was 73 and ranged from 
68 to 85. This suggests that all participants considered the 
tool better than average interfaces and that the tool has 
reasonable usability overall. The lowest scores related to a 
perception that the tool was unnecessarily complex. There 
was no difference between the different PtDA versions. 



Think aloud analysis 

In total, the think aloud analysis yielded 65 comments re- 
lating to user-experience. Positive comments were gener- 
ally around the interactive features of the tool and its ease 
of moving from step to step. The subject of negative com- 
ments included the amount of words required to read, the 
wording of key instructions and a lack of intuitiveness in 
how to interact with some features (such as the value 
clarification exercise). Overall, 16 out of 20 (80%) stated 
they had no major issues while using the tool. The 4 par- 
ticipants that suggested the tool was difficult to use were 
all in the oldest age quartile. Points of improvement in- 
cluded: provision of examples to show how to interact 
with key features (9 out of the 20 [45%]), clearer colours, 
speed of the software, and wording of certain questions 
(all less than 25% of participants). 



Usefulness 
Eye-tracking 

Regardless of the type of summary, we observed an order 
effect whereby respondents spent more time observing the 
attributes at the top of the list (23% of time spent on first 
attribute) versus bottom of the list (13% of the time spent 
on last attribute). This influenced the amount of time indi- 
viduals fixated on attributes they felt were more important 
to them (Table 2). In the conventional summary, 18% of fix- 
ation duration was spent on the most important attribute, 
followed by 16% of duration on the second most important 
attribute. The DCIDA summary demonstrated an increase 
to 30% and 18% respectively. Similarly, in the conventional 
display, 12% of time was spent on the least important attri- 
bute, compared to only 5% of time using the DCIDA. Ana- 
lyzing the subgroup of participants that changed their 
preferred option between the conventional and DCIDA 
summaries shows even greater differences in fixations 
(Table 2). The heatmaps in Figure 4 describe the influence 
of DCIDA on two individuals. 

Quantitative responses 

For the 12 participants using the OSA tool, based on 
the conventional display, 5 participants chose the Oral 

Table 2 Results of system usability scale, decisional 
conflict, and eye tracking 





Conventional 


DCIDA 


System Usability Score, mean (range) 




74 (68-85) 


Decisional conflict uncertainty 
subscale (0 = good, 4 = bad). 






1 am clear about the best choice for me 


2.9 


2.3 


1 feel sure about what to choose 


2,8 


2.2 


This decision is easy for me to make 


3,4 


3.1 


Overall score 


50.4 


38.3 


Mean fixation duration (sees) 






Most important attribute 


4,7 


5.6 


2"^^ most important attribute 


4,2 


3.3 


Least important attribute 


3,1 


1.1 


Other attributes 


14,1 


8.5 


Other areas of the screen 


16.2 


12.8 


Total 


42.3 


31.3 


Mean relative fixation 






Most important attribute 


18% 


30% 


2"^^ most important attribute 


16% 


18% 


Least important attribute 


12% 


6% 


Relative fixation - in 7 participants 
changed preference 






Most important attribute 


17% 


34% 


2"^^ most important attribute 


14% 


19% 


Least important attribute 


10% 


5% 
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Key 



<^ More visual attention less visual attention ~^ 



2l Example of respondent 8 who considered "Cost," "Apparatus," and "Reduction in 
Snoring" to be most important 

Conventional display with heatmap The DCIDyi display 





No Treatment 


Dental appliance 




Daytime sleepiness/fatigue 


No improvement 


Moderate improvement 


Return to normal-^ 


Risk of cardiovascular disease 


■4 in 100 


2 in 100 


1 in 100 


Reduction in snoring 


Remains the same 


^^^artially reduced 


Completely eliminated 


Apparatus 


None 


Relatively invasive 


Invasive and noisy 


Portability 


No issue 


No issue 


Limits travel 






Temporary side-effects 


None 


Mouth pain and dryness 


'ain and skin irritation 


Permanent side-effects 


None 


Change in bite 


None 


Cost 


$0 


52500 


$1500 




Which do you prefer? 








o 


O 


O 





No Treatment 


Dental appliance 




CPAP 


Cost 




$2500 






$1500 


Apparatus 


>B 


Relatively invasive 


Invasive and noisy 


Reduction in snoring 


Remains the same 


Partially reduced 


Completely eliminated 


Daytime sleepiness/fatigue 


No improvement 


Moderate improvement 


Return to normal 


Risk of cardiovascular disease 


4 in 100 




2 in 100 






1 in 100 


Permanent side-effects 


None 


Change in bite 
Mouth pain and dryness 
No issue 


None 

Pain and skin irritation 
Limits travel 


Temporary side-effects 
Portability 


None 
No issue 


Which do you prefer? 






O 


O 


O 



In the conventional PtDA, the individual spent 32% of their time looking at the attributes "Daytime sleepiness I fatigue " and 
"Risk of cardiovascular disease" (indicated in red), and spent 22% of their time considering the attributes of most importance to 
them, "Cost" and "Apparatus. " In the DCIDA, this individual switched to looking at their most important attributes 41% of 
time. 

b Example of Respondent 12 who considered "Permanent side-effects," 
"Sleepiness/fatigue," and "Portability" to be most important. 



Conventional display with heatmap 



The DCIDA display with 





No Treatment 


Dental appliance 










Daytime sleepiness /fatigue 


No improvement 


Moderate improvement 


Return to normal 


Risk of cardiovascular disease 


4 in 100 


2 in 100 


1 in 100 






Reduction in snoring 


^^ins the same 


Partially reduced 


Completely eliminat J 


apparatus 


None 


Relatively invasive 


Invasive and noisy 


Portability 


No issue 




Limits travel 


No issue 


Temporary side-effects 


None 


Mouth pain and dryness 


Pain and skin irritation 


Permanent side-effects 


None 


Change in bite 


None 


Cost 




$2500 


$1500 


■^1 


Which do you prefer? 


0 


0 


° 









No 1^^^ 


ent 


Dental appliance 








1 




I ' 


Permanent side-effects 


None 




Ch£^Hi^H 


None .A 












Daytime sleepiness/fatigue 


No improvement 


Moderate improvement 


Return to normal 


Portability 


No issu 




No issue 


Limits travel 










Apparatus 

Reduction in snoring 

Risk of cardiovascular disease 


None 
Remains the same 
4 in 100 


Relatively invasive 
Partially reduced 
2 in 100 


Invasive and noisy 
Completely eliminated 
1 in 100 


Cost 


$0 


$2500 


$1500 


Temporary side-effects 


None 


Mouth pain and dryness 


Pain and skin irritation 


Which do you prefer? 


O 






0 


O 



Figure 4 Heatmaps of 2 examplar respondents (where colour represents the proportion of time spent fixating in areas within the 
defined cell space). 



Appliance while 6 chose CPAP. Two respondents were 
'very sure' of their decision, while 4 respondents were 
'not very sure,' with the rest being 'moderately' or 'some- 
what sure.' When presented with the DCIDA version of 
the summary information, 4 of the 12 participants chan- 
ged their preferred option. The results for the cancer 
and osteoarthritis tools produced similar results with 2 
participants changing their decision for the cancer tool, 



1 for the osteoarthritis tool. Overall, the decisional conflict 
uncertainty subscore was 50.4 for the conventional sum- 
mary, reducing to 38.3 in the DCIDA version (Table 2). 

Th'mk aloud analysis 

Of the 45 comments coded for usefulness, 28 (62%) were 
positive. The predominant positive themes were that the 
treatment information was easy to access and the DCIDA 
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summary information either confirmed or improved their 
treatment choice. Negative comments were firom individ- 
uals who felt that they knew their decision and were frus- 
trated that they had to negotiate all the steps before 
indicating their preference option. Four of the participants 
suggested that they felt 'nudged' to one of the choices, in a 
positive frame. When prompted for further clarity, it was 
not clear whether they felt this had encroached their au- 
tonomy to choose or not. 

Discussion 

This explorative study investigated the feasibility of 
using nudges based on MCDA to improve the quality of 
treatment decisions. Software was developed to enable 
information from conventional PtDAs to be restructured 
in how it is presented to the user through dynamic and 
interactive interfaces. The software was tested for both 
usability and usefulness in 20 participants. Both aspects 
of user testing provided some positive results, and pro- 
vide important information for future development. 

There is limited research on the use of behavioural eco- 
nomic approaches to improve patients' use of PtDA. A 
study by Ubel et al. found that order effects in decision 
aids could be unbiased by providing the patient further in- 
formation (in graphical form) [10]. This is one approach 
for encouraging System 2 thinking, yet a concern with tar- 
geting numerous biases through this general technique is 
that providing more information can sometimes over- 
whelm people, causing them to revert back to System 1 
thinking [54]. The DCIDA approach has sought to enable 
users to read less information, but focus on information 
that will most likely influence their choice. 

While there has been substantial attention to 'nudge' 
theory in health [55,56], to our knowledge this theory has 
not been tested in PtDAs. Default options have become 
the predominant 'nudge' used in health interventions to 
date [57], but have typically selected a single default option 
for all users. For example, organ donation programs may 
'nudge' patients to enrol by making organ donation the 
default option. It has been proposed that nudges could be 
used in PtDAs for conditions where the evidence clearly 
indicates that one treatment option is superior to the 
others [58]. This is controversial as most PtDAs are devel- 
oped for preference sensitive decisions where two or more 
medically appropriate options exist, and they seek to pro- 
mote rather than diminish patient autonomy. DCIDA has 
been designed as a bridge between non prescriptive PtDAs 
and overdy prescriptive decision analysis tools. 

The objective of this study was to examine if there was a 
difference in response between the two versions of PtDAs. 
If no difference was observed, we would reject the hypoth- 
esis that the DCIDA version had any impact. While we es- 
tablish some preliminary demonstration of effectiveness, 
this study alone cannot ascertain whether the impact is 



real or useful and should be interpreted with caution for 
three primary reasons. First, participants considered the 
conventional summary before the DCIDA version, there- 
fore an ordering effect might have been observed whereby 
they became more informed as they spent more time 
viewing the information. On average 42 seconds was spent 
viewing the conventional display versus 31 seconds on the 
DCIDA display. It is also difficult to disentangle the effects 
of each aspect of the DCIDA version that differed from 
the conventional summary, such as the values clarification 
exercise, the layout, or colour hues. While it would be un- 
feasible to investigate the impact of each design feature, 
we have subsequently evaluated the impact of ordering ef- 
fects in a larger controlled study [59] . A second limitation 
relates to putting the values clarification exercise at the be- 
ginning of the decision aid. We deliberately presented this 
exercise up front to elicit more stable values from partici- 
pants. However, recent studies have suggested that it can 
be problematic to engage in importance weighting too 
soon in the decision- making process [60,61]. A related 
third reason is that we do not know if participants who 
changed their decision actually made an improved choice. 
This challenge of measuring the quality of patients choices 
is a limitation in all research on PtDAs [62] . 

Further, we acknowledge some limitations to our use of 
standard measures. With regard to the System Usability 
Scale (SUS), this measure has been used frequentiy for us- 
ability evaluations of Internet-based interfaces; it is not 
used typically for evaluations of Internet-based PtDAs. 
Validation of the scale was based on studies of interfaces 
up to 18 years old and, thus, there are contextual and de- 
sign differences between DCIDA and the average tools 
used to validate the SUS. However, by triangulating the re- 
sults of our SUS scores with our think aloud and eye- 
tracking results, we determined that the initial prototype 
has acceptable usability, though we aim to improve it in 
future iterations of the prototype and to use and report a 
newer iteration of the SUS developed by Bangor et al. [63] . 
Additionally, we chose to use the Uncertainty subscale of 
the Decisional Conflict Scale and acknowledge that our 
analysis would have benefitted from also using the Values 
Clarity subscale [49] . Use of this additional measure would 
have contributed to our understanding of how DCIDA 
impacted participants' ability to arrive at stable values. 

While the usability scores and improvements in deci- 
sional conflict are encouraging, they suggest there is still 
opportunity to further improve the tool. At the time data 
collection, the DCIDA software was in alpha stage, and 
the results of this research have motivated us to move to a 
different platform for the beta version. The higher usabil- 
ity results may also be due to the hypothetical nature of 
the task. Participants did not have the diseases and were 
aware they were testing a tool. In addition, subjects were 
majority college-educated who had access to and comfort 
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using computers. These findings may not be generalizable 
to a more heterogeneous population with lower education 
or computer proficiency. We are also cognisant that it will 
be important for the software to be compatible with Inter- 
net use on tablets, which will require separate testing. 

Given these opportunities for future research, we plan 
to further explore the influence of the DCIDA in subse- 
quent studies. In these, individuals will be randomized be- 
tween a conventional PtDA and a DCIDA version and 
plan to determine if DCIDA's unique design features lead 
to improvements in decision quality, including concord- 
ance between what matters most to individual patients 
and their chosen option. We propose to consider carefully 
how value concordance is measured. There are no stand- 
ard criteria for studying values concordance and a recent 
Cochrane review [1] shows that there is substantial het- 
erogeneity among the measures that authors have used to 
date. We agree with the growing number of researchers 
calling for further study into the "active ingredients" of 
values clarification [64] and the creation of standard mea- 
sures for analyzing values congruence [62,65]. Such re- 
search will assist us in identifying what proportion of 
people make values congruent decisions when they use 
DCIDA in comparison to conventional tools. We also plan 
to ask questions about patients' attitudes to the role of 
nudges in making autonomous decisions [38]. Finally, we 
believe it will be crucial to include patients of varying 
health literacy, and numeracy to examine the influence of 
the tool in different patient groups. 

Conclusion 

The DCIDA has been developed to enhance conventional 
PtDAs to assist patients in choosing the treatment that is 
most congruent with their informed values. This paper re- 
ports on the theoretical motivation for the DCIDA and 
then describes an experiment in which the tool is user 
tested. The results give some empirical support that the 
DCIDA is understandable to users and that it can help 
users focus on attributes that are of individual importance 
to them - to the extent that some participants changed 
their decisions. A number of valuable insights were learned 
for improving the next version of the DCIDA. In conclu- 
sion, we propose that the DCIDA is a promising approach 
to improve conventional PtDAs. Further development is 
required to improve its usability and usefulness; however 
research on testing preliminary effectiveness on patient 
decision-maldng is justified. 
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